Fraud detection in interactive voice response systems

ABSTRACT

Systems and methods for call detail record (CDR) analysis to determine a risk score for a call and identify fraudulent activity and for fraud detection in Interactive Voice Response (IVR) systems. An example method may store information extracted from received calls. Queries of the stored information may be performed to select data using keys, wherein each key relates to one of the received calls, and wherein the queries are parallelized. The selected data may be transformed into feature vectors, wherein each feature vector relates to one of the received calls and includes a velocity feature and at least one of a behavior feature or a reputation feature. A risk score for the call may be generated during the call based on the feature vectors.

The present application claims priority to U.S. Provisional PatentApplication Ser. No. 62/241,478, filed Oct. 14, 2015, and U.S.Provisional Patent Application Ser. No. 62/371,103, filed Aug. 4, 2016,the entire disclosures of which are hereby incorporated by reference.

BACKGROUND

Telephone-based systems collect useful metadata, including Call DetailRecords (CDR). By analyzing large volumes of CDR data from a phonesystem, behavioral patterns can be discerned that can identifyfraudulent activity.

It is desirable to identify fraudulent activity during a call ordetermine a risk score for a call during the call. Waiting until afterthe call is completed may mean a fraud is already perpetrated or anopportunity to acquire information regarding the perpetrator has beenlost. Detecting fraud during a call allows the caller to actaccordingly. For example, in addition to disallowing access to anaccount, the caller may request additional information about the callerto supplement a profile concerning the caller.

Deployment of a system to identify fraudulent activity during a call orto determine a risk score for a call during the call may result in therapid acquisition of many records. For example, a bank may have millionsof customers or users. Performing look-up, detecting fraud, and takingaction in response to the fraud detection (or determining lowprobability of fraud and taking action in response to the lowprobability of fraud) during a call becomes increasingly difficult asthe number of records increases, both in terms of the number of usersand the number of records per user. Some features relevant todetermining a risk score may be difficult to determine rapidly.

As recognized by the inventors, data contained in CDRs and data obtainedduring a call may be used to determine fraud. Using the systems andmethods disclosed herein, fraudulent activity may be detected andidentified, and a risk score determined, during a call, including insituations involving large amounts of data. Further, the risk score fora call may be determined during the call using features which may becomplex, including behavior, reputation, and velocity features. Amachine learning model, which may include a supervised classifier, maybe used in the determination of the risk score of a call during thecall.

Interactive voice response (IVR) systems are widely used as a means forappropriately routing a call into a call center. As the use of thesesystems has become more widespread, users of the IVR have been able toaccomplish an ever-increasing number of tasks from within the system,often without interacting with a human. These tasks include accountauthentication, balance checking, and more. With this increase incapability in the IVR has come an increase of malicious activitydirected towards extracting information and committing fraud by takingadvantage of weaknesses inherent in an automated system.

As recognized by the inventors, user behavior in an IVR system mayenable the detection of fraudulent or malicious interactions whencompared to the behavior of genuine users. By monitoring how usersinteract with a system historically, a detection framework may beconstructed to predict and prevent fraudulent or malicious activity.

SUMMARY

This specification describes technologies relating to CDR analysis toidentify fraudulent activity and fraud detection in IVR systems.

In general, one aspect of the subject matter described in thisspecification can be embodied in a computer-implemented method or anapparatus.

According to an embodiment, the invention may include acomputer-implemented method for determining a risk score of a callreceived by an Interactive Voice Response (IVR) system, thecomputer-implemented method comprising: creating a feature vector basedon interaction with the IVR system during the call; and using a machinelearning model to determine the risk score of the call based on thecreated feature vector.

According to other aspects of the invention, the computer-implementedmethod may further comprise: representing prior calls as featurevectors; labeling the feature vectors representing the prior calls; andtraining the machine learning model using the labeled feature vectors.

According to other aspects of the invention, information regarding thecall is stored in a database before the using a machine learning modelto determine the risk score of the call based on the created featurevector.

According to other aspects of the invention, information regarding thecall is retrieved from the database before the using a machine learningmodel to determine the risk score of the call based on the createdfeature vector.

According to other aspects of the invention, the created feature vectoris based on a behavior feature of the call, and wherein the behaviorfeature of the call is a sequence of actions taken during interactionwith an IVR system.

According to other aspects of the invention, the created feature vectoris based on a behavior feature of the call; and the behavior feature ofthe call is at least one of an action taken during interaction with anIVR system, an amount of time elapsed between actions taken duringinteraction with an IVR system, providing incorrect information to anIVR system, a number of times a specific activity in an IVR wasperformed, a number of times an IVR was called during a measure of time,a volume or a duration of at least one DTMF tone during interaction withan IVR system, an amount of time elapsed between DTMF tones, a use ofvoice during interaction with an IVR system, an amount of time elapsedbetween a beginning of an IVR prompt and a user's spoken response to theIVR prompt, or an amount of time elapsed between an IVR prompt and anaction taken during interaction with an IVR system.

According to other aspects of the invention, information regarding theprior calls is stored in a non-relational database.

According to other aspects of the invention, the prior calls are allfrom a same phone number.

According to other aspects of the invention, the prior calls are from aplurality of phone numbers, and each of the labeled feature vectors usedto train the machine learning model has a same label as each of theother labeled feature vectors used to train the machine learning model.

According to other aspects of the invention, the labeling the featurevectors representing the prior calls is done using a first labelrepresenting genuineness and a second label representingnon-genuineness.

According to other aspects of the invention, the representing priorcalls as feature vectors includes representing interaction with an IVRsystem using action words.

According to other aspects of the invention, the representing priorcalls as feature vectors includes: representing interaction with an IVRsystem using action words in a document; and converting the document toa feature vector.

According to other aspects of the invention, the representing priorcalls as feature vectors includes representing interaction with an IVRsystem using at least one of N-grams or skip N-grams.

According to other aspects of the invention, the representing priorcalls as feature vectors includes determining an inverse documentfrequency.

According to other aspects of the invention, the computer-implementedmethod may further comprise taking an action based on the risk score ofthe call.

According to other aspects of the invention, the taking an action basedon the risk score of the call includes at least one of displaying therisk score on a display during the call, storing the risk score in adatabase during the call, altering an IVR call flow during the call,notifying police, or notifying an owner of an IVR system.

According to other aspects of the invention, the taking an action basedon the risk score of the call includes at least one of altering an IVRcall flow during the call, disabling a feature in the IVR system duringthe call, locking down the IVR system, requiring alternativeidentification during the call, or requesting additional informationduring the call.

According to other aspects of the invention, the interaction with theIVR system includes responses to the IVR system represented using actionwords.

In another embodiment of the invention, the invention may include acomputer-implemented method for determining a risk score for a call, thecomputer-implemented method comprising: storing information extractedfrom received calls; performing queries of the stored information toselect data using keys, wherein each key relates to one of the receivedcalls, and wherein the queries are parallelized; transforming theselected data into feature vectors, wherein each feature vector relatesto one of the received calls and includes a velocity feature and abehavior feature; and generating, during the call, the risk score forthe call based on the feature vectors.

According to other aspects of the invention, each feature vectorincludes a reputation feature.

According to other aspects of the invention, the computer-implementedmethod may further comprise: training a machine learning model using thefeature vectors; using the machine learning model to generate the riskscore for the call; and displaying the risk score for the call on adisplay during the call, wherein the queries are parallelized using athread pool.

In another embodiment of the invention, an apparatus that determines arisk score of a call received by an Interactive Voice Response (IVR)system may comprise at least one processor; and a non-transitorycomputer-readable medium coupled to the at least one processor havinginstructions stored thereon that, when executed by the at least oneprocessor, causes the at least one processor to: create a feature vectorbased on interaction with the IVR system during the call; and use amachine learning model to determine the risk score of the call based onthe created feature vector.

According to other aspects of the invention, the created feature vectoris based on a behavior feature of the call, and the behavior feature ofthe call is a sequence of actions taken during interaction with an IVRsystem.

In another embodiment of the invention, an apparatus that determines arisk score for a call may comprise at least one processor; and anon-transitory computer readable medium coupled to the at least oneprocessor having instructions stored thereon that, when executed by theat least one processor, causes the at least one processor to: storeinformation extracted from received calls; perform queries of the storedinformation to select data using keys, wherein each key relates to oneof the received calls, and wherein the queries are parallelized;transform the selected data into feature vectors, wherein each featurevector relates to one of the received calls and includes a velocityfeature and a behavior feature; and generate, during the call, the riskscore for the call based on the feature vectors.

In another embodiment of the invention, the invention may include acomputer-implemented method for determining a risk score for a call, thecomputer-implemented method comprising: receiving a call from aparticular phone number; retrieving pre-stored information relating tothe particular phone number to derive a reputation feature and avelocity feature; including the reputation feature and the velocityfeature in a feature vector; and generating a risk score for the callbased on the feature vector.

According to other aspects of the invention, the computer-implementedmethod may further comprise: labeling the feature vector; training amachine learning model using the labeled feature vector and otherlabeled feature vectors; and using the machine learning model togenerate the risk score for the call.

According to other aspects of the invention, the computer-implementedmethod may further comprise: taking an action based on the risk scorefor the call, wherein the taking an action based on the risk score forthe call includes at least one of displaying the risk score on a displayduring the call, storing the risk score in a database during the call,altering an interactive voice response (IVR) call flow during the call,notifying police, notifying an owner of an IVR system, disabling afeature in an IVR system during the call, locking down an IVR systemduring the call, requiring alternative identification during the call,or requesting additional information during the call.

According to other aspects of the invention, the feature vector includesa behavior feature derived from the call.

According to other aspects of the invention, the pre-stored informationis stored in a database and retrieved from the database before theincluding the reputation feature and the velocity feature in the featurevector.

According to other aspects of the invention, the velocity feature is asequence of calls or attempted calls from at least one originating phonenumber similar to the particular phone number.

According to other aspects of the invention, the velocity feature is atleast one of a number of distinct account identifiers, a number ofdistinct originating phone numbers associated with an accountidentifier, or a number of destinations called.

According to other aspects of the invention, the feature vector includesa velocity feature based on at least one of a number of calls, aduration of at least one prior call, a duration between calls, or aperiodicity between calls.

According to other aspects of the invention, the reputation feature isat least one of suspicious activity, malicious activity, a priorcomplaint, a device type, a carrier, a route taken by the call prior toentering a telephone exchange, a route taken by the call after leaving atelephone exchange, or a location.

According to other aspects of the invention, the pre-stored informationis stored in a non-relational database.

According to other aspects of the invention, the pre-stored informationis stored in a graph database.

According to other aspects of the invention, the risk score is generatedduring the call.

According to other aspects of the invention, the retrieving pre-storedinformation relating to the particular phone number to derive areputation feature and a velocity feature is done using at least onequery to select the pre-stored information.

In another embodiment of the invention, the invention may include acomputer-implemented method for determining a risk score for a call, thecomputer-implemented method comprising: storing information extractedfrom received calls; performing queries of the stored information toselect data using keys, wherein each key relates to one of the receivedcalls, and wherein the queries are parallelized; transforming theselected data into feature vectors, wherein each feature vector relatesto one of the received calls and includes a velocity feature and areputation feature; and generating, during the call, the risk score forthe call based on the feature vectors.

According to other aspects of the invention, each feature vectorincludes a behavior feature.

According to other aspects of the invention, the computer-implementedmethod may further comprise: training a machine learning model using thefeature vectors; using the machine learning model to generate the riskscore for the call; and displaying the risk score for the call on adisplay during the call, wherein the queries are parallelized using athread pool.

In another embodiment of the invention, an apparatus that determines arisk score for a call may comprise the following: at least oneprocessor; a non-transitory computer readable medium coupled to the atleast one processor having instructions stored thereon that, whenexecuted by the at least one processor, causes the at least oneprocessor to: receive a call from a particular phone number; retrievepre-stored information relating to the particular phone number to derivea reputation feature and a velocity feature; include the reputationfeature and the velocity feature in a feature vector; and generate arisk score for the call based on the feature vector.

According to other aspects of the invention, the velocity feature is asequence of calls or attempted calls from at least one originating phonenumber similar to the particular phone number.

According to other aspects of the invention, the apparatus may furthercomprise a display that displays, during the call, the risk score forthe call.

In another embodiment of the invention, an apparatus that determines arisk score for a call may comprise: at least one processor; anon-transitory computer readable medium coupled to the at least oneprocessor having instructions stored thereon that, when executed by theat least one processor, causes the at least one processor to: storeinformation extracted from received calls; perform queries of the storedinformation to select data using keys, wherein each key relates to oneof the received calls, and wherein the queries are parallelized;transform the selected data into feature vectors, wherein each featurevector relates to one of the received calls and includes a velocityfeature and a reputation feature; and generate, during the call, therisk score for the call based on the feature vectors.

The details of one or more embodiments of the invention are set forth inthe accompanying drawings which are given by way of illustration only,and the description below. Other features, aspects, and advantages ofthe invention will become apparent from the description, the drawings,and the claims. Like reference numbers and designations in the variousdrawings indicate like elements.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a table showing an IVR call flow for a normal (benign orgenuine) user.

FIG. 2 is a table showing a phishing attempt in an IVR call flow.

FIG. 3 is a block diagram illustrating a system for determining a riskscore of a call according to one or more embodiments of the presentinvention.

FIG. 4 is a block diagram illustrating a method for determining a riskscore of a call received by an Interactive Voice Response (IVR) systemaccording to one or more embodiments of the present invention.

FIG. 5 is a block diagram illustrating a method for generating a machinelearning model which may be used in determining a risk score of a callaccording to one or more embodiments of the present invention.

FIG. 6 is a block diagram illustrating an example computing device.

FIG. 7 is a block diagram illustrating a method for determining a riskscore of a call according to one or more embodiments of the presentinvention.

FIG. 8 is example code using the Aggregation application programminginterface (API) with PyMongo according to one or more embodiments of thepresent invention.

FIG. 9 is example code to store feature vectors of calls to a databaseusing PyMongo according to one or more embodiments of the presentinvention.

FIG. 10 is a block diagram illustrating a method for generating featuresaccording to one or more embodiments of the present invention.

FIG. 11 is a block diagram illustrating a method for generating featuresaccording to one or more embodiments of the present invention.

FIG. 12 is a table showing call flows for a first call and a secondcall, both calls from the same phone number, wherein the two calls are aphishing attempt.

FIG. 13 shows an example of IVR reconnaissance performed by a frauddetection system.

FIG. 14 shows an example of IVR reconnaissance performed by a frauddetection system.

FIG. 15 describes a method for generating IVR call flow features andusing the IVR call flow features to train a machine learning model whichmay be used to classify calls according to one or more embodiments ofthe present invention.

FIG. 16 is a drawing illustrating a graph of calls and components of agraph which may be used in one or more embodiments of the presentinvention.

FIG. 17 is a chart of feature vectors for calls plotted in a featurespace comprising call duration and an automatic number identification(ANI) scanning feature, with a region indicating fraudulent callsaccording to one or more embodiments of the present invention.

FIG. 18 is a block diagram illustrating a method for determining a riskscore of a call according to one or more embodiments of the presentinvention.

FIG. 19 is a block diagram illustrating a method for determining a riskscore of a call according to one or more embodiments of the presentinvention.

DETAILED DESCRIPTION

CDR analysis for detecting fraudulent activity and fraud detection inthe IVR may involve the following: designing features from CDR data or auser's interaction with the IVR to represent each call as a featurevector, using a labeled set of feature vectors to train a machinelearning model, and using the trained model for scoring the riskiness ofeach call. In at least one embodiment, a user's interaction with the IVRduring a single call may be represented in an IVR call flow or a calldetail record. A CDR may include data regarding the call in addition todata relating to the user's interaction with the IVR.

Characteristics of CDR Data

A typical CDR data set has at least one record for each phone call. Theexact schema of the CDR records vary widely depending on the domain,however, most of them contain at least the following: start timestampand, if the call has ended, end timestamp, the originating telephonenumber (originating phone number) of the call from a source ororiginating automatic number identification (ANI) of the call, thedestination ANI of the call if applicable, the route taken by the callentering an exchange, the route taken by the call leaving an exchange,whether the call successfully connected, and the call type (voice, SMS,MMS, etc.).

Application-specific CDR information may include account numbers orother information pertaining to a called party running an application.For example, a bank might have an application that tracks whichaccount(s) a caller tried to access in the call. Application-specificCDR information may also include IVR information. For example, asophisticated CDR system can create additional records per call,tracking the individual steps that a caller traverses in the IVR.Further, application-specific CDR data may include the amount of timeelapsed between steps (whether IVR prompt or caller response) in an IVRcall flow, including the time between key presses or DTMF tones.Application-specific CDR information may also include locationinformation. For example, mobile networks may track base station andcell tower information for a call and include the information in the CDRfor the call.

All of this data represents a rich source of information for inferringpatterns in user behavior. Finding these patterns, however, requiresrepresenting the data in creative ways so that features of interest maybe visualized and extracted. For example, graphs may be used torepresent this data.

Representing CDR Data Using Graphs

A graph is essentially a network with nodes connected by edges. Edgescan be directed or undirected. Both nodes and edges can be annotatedwith information. While they are simple to define, graphs are a powerfulway to represent heterogeneous pieces of information which are relatedto each other, and to understand complex interactions between them.

FIG. 16 is a drawing illustrating a graph of calls and components of agraph which may be used in one or more embodiments of the presentinvention. For example, CDR data may be represented in a graph (1600)with source (1610) and destination (1630) ANIs as nodes. The edges(1620) in the graph may be directed, representing a call made from thesource to the target. The edges (1620) can be annotated with informationabout the number, duration, and timings of the calls. The nodes (1610)and (1630) can be annotated with account numbers and other userinformation.

A graph can show interesting properties like node degree, clustering,and connectivity. The graph may represent activity over a finite lengthof time. With respect to node degree, it may be observed that mostsource ANIs call 1-3 destination ANIs. However, a small number of ANIscall a large number of destinations, e.g. spammer telemarketers. In agraph where each node represents either a source ANI or a destinationANI, ANIs calling a large number of destinations would be represented bynodes having high degree. These exceptions might represent some form offraud including “ANI scanning”, which may be relevant in determiningpremium rate fraud as discussed below.

With respect to clustering and connectivity, it may be observed that thegraph is clearly split into many small and a few large connectedcomponents. The question arises as to whether some connected componentsof unusual size may represent fraud. Traffic pumping may involve usingas much time as possible during a call to a toll-free number and/ormaking an unusually high number of calls to a toll-free number. A badactor's purpose in traffic pumping may be to increase the fees for thecaller's local exchange carrier, which may be entitled to intercarriertransfer fees from the called party's telecommunications provider.Traffic pumping may be performed by a large number of ANIs in order toevade detection. Thus, a toll-free number receiving calls from anunusual number of ANIs represented by a destination node (1630) havingunusually high fan-in (number of in-edges) might indicate trafficpumping.

These observations about graphs may be represented as numeric valuesthat can be used as features in further statistical and machine learninganalysis.

IVR Systems

Typically an IVR system will utilize dual-tone multi-frequency (DTMF)signaling to transmit the user's selections and information through thenetwork. DTMF signaling is a signaling system used by telephonyinfrastructure to communicate infoitnation. When a user dials digits onthe keypad of a phone or other telephony device, the “touch tones” areDTMF.

As traditional public switched telephone network (PSTN) traffic isconverted to voice over IP (VoIP), the actual audio tones generated bythe device, are often converted to an encoding format specified in RFC4733. This conversion removes almost all of the audible tone from thecall, and replaces it with a digital representation of the digit thatwas pressed, the duration of the tone, and a few other pieces ofinformation.

Often, telephony equipment that participates in the connection andhandling of a call will interpret the RFC 4733 encodings of the keypresses and respond accordingly. This is what the IVR uses to gatherinformation from the user after prompting the user to enter informationlike account number, date of birth, social security number, etc. When auser enters this sort of information, and it is converted to RFC 4733,the IVR system simply needs to read the digit pressed out of the callstream as opposed to “listening” for tones and determining thecorresponding numbers.

An IVR system may utilize the user's voice to transmit the user'sselections and information to the IVR system using conventional speechrecognition.

Feature Design for CDR Analysis

Features, in machine learning terminology, are metrics which,statistically speaking, are likely to distinguish instances. Forexample, “good” (e.g. not fraudulent or likely not fraudulent) instancesmay be distinguished from “bad” (e.g. fraudulent or likely fraudulent)instances. In the case of CDRs, reputation, velocity, and behaviorfeatures may be extracted.

Reputation Features

When a phone call is received, it may be useful to know what is knownabout the reputation of the calling phone number and whether it hasalready been associated with suspicious or malicious activity. Severalpieces of relevant information may be tracked to form reputationfeatures including, without limitation, the following: carrier, devicetype (e.g. landline, cell, VoIP, softphone, or specific phone model),and complaints associated with the ANI/calling party's phone number,both in public and private databases. Using these and other data a riskscore can be calculated for each phone number.

Velocity Features

Velocity features summarize the trends associated with an ANI over time.Many velocity features emerge naturally from representing the CDR dataas a graph. In addition to representing ANIs as nodes and calls asundirected edges or directed edges from source to destination ANI, otherinformation such as accounts, names, and locations may also beincorporated into the graph. Some typical and useful velocity featuresinclude the following: the number of destination ANIs that a source ANIhas called, the number of user accounts associated with an ANI and viceversa, the average frequency of calls from or to an ANI, and the averageinterval between subsequent calls from or to an ANI. These features areparticularly useful when it is suspected that specific actions areassociated with malicious activity. Many of these features areessentially graph properties: node degree, edge creation frequency, sizeof connected components, and so on.

Behavior Features

Behavior features indicate the genuineness or suspiciousness of acaller's actions, especially with respect to a caller's interaction withan IVR system. In a modern IVR system, information can be gathered aboutuser behavior to distinguish their navigation or information entry asgenuine or malicious. In order to make these insights, multiple aspectsof the IVR system are used, including but not limited to DTMF signalinginformation like volume and duration of a DTMF tone and the amount oftime elapsed between DTMF tones, voice navigation, caller metadata, andthe entries made by the user (noted as behavior features). Using theseand other data, a risk score can be calculated for each phone number.

Each user of an IVR system has a unique way of navigating within theoptions and accomplishing their desired outcome. It has been recognizedand verified that based on behavior features, genuine individual usersand genuine users as a group have “fingerprints”, including patterns ofuse and activity, different than those of malicious users. Maliciousactivity includes, but is not limited to, brute force attempts,information mining (reconnaissance), robotic dialing, spoofing, hidingbehind gateways, and account takeover.

By tracking statistical information for behavior features, a frameworkfor fraud detection can be constructed to predict and prevent maliciousactivity, including malicious activity in the IVR. In addition, afeedback loop confirming fraud or malicious activity enhances thesystem's ability to detect and deter future attempts. The feedback loopmay comprise retraining a machine learning model using newly labeleddata and/or data with revised labels.

The IVR call flow sequence may be broken up into chunks, and each chunkmay then be converted into a numeric feature vector. The chunks may beof different sizes. These feature vectors may have a very high number ofdimensions. In at least one embodiment, features that are especiallypredictive of the label of interest, e.g., “fraud” or “not_fraud” or“human” or “robot”, may be selected using a feature selection techniquesuch as the chi-squared method.

Using Text Analysis to Represent IVR Call Flow as Features

A typical IVR call flow may be represented in CDR data as a sequence ofrecords, each of which has a timestamp, along with the name of the stepand information about its outcome. FIG. 1 is a table 100 showing an IVRcall flow for a normal (benign or genuine) user. FIG. 1 depicts a singlefailed PIN attempt, most likely a mistake, followed by a successful PINentry and a balance check.

On the other hand, FIG. 2 is a table 200 showing a possible phishingattempt. The user tries several times to enter a PIN, fails, and thentries another authentication method and succeeds in accessing theaccount. After checking the account balance, the user attempts to takeover the account by changing the PIN.

Tables 100 and 200 are just two example representations of an IVR callflow and are not intended to limit the forms of representation of IVRcall flow or indicia of fraud. Information from IVR call flow which maybe recorded or encoded as features may also include entries made by auser (whether using a keypad, DTMF tones, teletype, teletypewriter, TTYdevice, the user's voice, or any other input device used by the user toprovide information to the IVR system) or the time the entry was made,perhaps with granularity more specific than just hours and minutes (e.g.seconds or milliseconds).

Represent One or More Call Flow Sequence(s) as a Document

A call flow sequence may be represented as text, including words in anatural language like English or a concatenation thereof. As usedherein, “text document” refers to the text representation of the callflow sequence which may be contained within a number of file formats,including a text document, a file containing English words or theconcatenation thereof in an ASCII encoding, a data file, a filecontaining data legible to a database management system, or a filecontaining information that may be queried using a relational database,a non-relational database, or a graph database.

In order to represent a call flow sequence as text, in at least oneembodiment of the present invention, each IVR step is transformed to adescription comprising one or more “action words”, and then a documentrepresenting the IVR call flow is created. For example, the IVR callflow sequences in FIGS. 1 and 2 could be represented using action wordsas follows: [LanguageMenu AccountEntry PinEntry PinEntry BalanceCheck]and [LanguageMenu AccountEntry PinEntry PinEntry PinEntry SSNEntrySSNEntry SSNEntry BalanceCheck PINChange], respectively.

Different representations of each step in the IVR call flow arepossible. For example, in FIG. 1, the first row having timestamp of 7:39am could be represented, without limitation, using one or more of thefollowing action words: “LanguageMenu”, “7:39 am”, “Success”,“LanguageMenuSuccess”, “English”, “LanguageMenuEnglish”, or“EnglishSuccess”, where English may have been a language selected by theuser. Action words may specify values entered by the user. Manycombinations of data may be represented using action words. Moregenerally, each IVR step may be represented using a descriptioncontaining information that has predictive value or leads to betterprediction.

One or more action words are used to provide a description of each IVRcall flow step, and the description contains information that may beuseful in determining a risk score. By including a timestamp for a stepin an IVR call flow, or an amount of time elapsed between steps in anIVR call flow, action words may be used in determining an amount of timeelapsed between actions taken during interaction with an IVR system. Byincluding a timestamp both for an IVR prompt and a user's response tothe IVR prompt, action words may be used in determining an amount oftime elapsed between an IVR prompt and an action taken duringinteraction with an IVR system. Queries may be performed to select databased on timestamp with a key such as caller phone number.

Call flow sequences from multiple calls, perhaps from the same source,may be combined to create a single document. The multiple calls may becontiguous calls. For example, FIG. 2 may represent one call made at7:39 am and lasting through 7:44 am. However, FIG. 12 may represent twocalls from the same originating ANI, the first call made at 7:39 am andthe second call made at 7:54 am. Instead of representing the two callsdepicted in FIG. 12 using two text documents, the two calls may berepresented as a single text document as follows: [LanguageMenuAccountEntry PinEntry PinEntry PinEntry SSNEntry SSNEntry LanguageMenuAccountEntry SSNEntry BalanceCheck PINChange]. Again, various forms ofaction words are possible. Each action word may have an associatedtimestamp. Generally, a number c>1 of calls may be represented as asingle text document by appending representations of the call flows asdescribed herein. The c calls may be from the same ANI. The c calls maybe from the same entity calling from more than one ANI. The c calls maybe from more than one ANI. The c calls may be from more than one entity.The c calls may be selected according to a time window in which the ccalls were made. The c calls may be selected according to an amount oftime elapsing between calls.

By using action words to represent a user's identity or ANI, a number oftimes an IVR was called by a particular user or particular ANI may berepresented. Further, by using action words comprising a timestampindicating the time a call was received, a number of times an IVR wascalled during a measure of time may be represented. Thus, by combiningaction words representing a user's identity or an ANI with action wordscomprising a timestamp indicating the time a call was received, a numberof times an IVR was called by a particular user or a particular ANIduring a measure of time may be represented. Queries may be performed toselect timestamp data with a key such as caller phone number.

Create Feature Vectors from Documents

Once call flow sequences are represented as documents, documents may beconverted to feature vectors. For this, in at least one embodiment ofthe present invention, a standard vector space model commonly used intext analysis may be used. In at least one embodiment of the presentinvention, the vector space model may work by processing the documentsthrough the following steps. First, break all the documents into terms,where each term may be an N-gram (a contiguous chunk of words up tolength N) or a skip N -gram (a non-contiguous chunk of words, typicallyomitting words of little predictive value). Second, calculate thefrequency of each term (TF) and the inverse document frequency of eachterm (IDF). Third, create a “term-document” matrix populated with TF-IDFvalues, wherein the matrix comprises m rows and n columns. Each of the ncolumns of the term-document matrix corresponds to a document, whereineach document contains call flow sequences from one or more calls. Thus,each column of the term-document matrix is an m -dimensional featurevector corresponding to a document. Each row of the term-document matrixcorresponds to a feature. Optionally, in order to reduce dimensionality,select the k features (k<m) most predictive of a label of interest usinga feature selection criterion, e.g. chi-squared. Finally, use thek-dimensional feature vectors (or the m-dimensional feature vectors) totrain a classifier to label calls using the label of interest.

Denote the frequency of term i in document j as tf_(i,j) and the inversedocument frequency of term i as idf_(i). Let n equal the total number ofdocuments in the collection and d_(i) equal the number of documents inthe collection that contain term i. Then, in at least one embodiment,define

${idf}_{i} = {\log \; \left( \frac{n}{d_{i}} \right)}$

and tf-idf_(i,j)=tf_(i,j)×idf_(i).

FIG. 15 describes a method for generating IVR call flow features andusing the IVR call flow features to train a machine learning model whichmay be used to classify calls according to one or more embodiments ofthe present invention. First, IVR actions taken during calls arerepresented (1510) as action words in documents. Second, action words inthe documents are converted (1520) to terms. Third, a frequency of eachterm in each document and an inverse document frequency of each term aredetermined (1530). Fourth, a term-document matrix is determined (1540)for the documents. Fifth, features from the term-document matrix areselected (1550) using a feature selection criterion. Sixth, a machinelearning model is trained (1560) using the selected features.

Case Study: State Benefits Provider

A major state benefits provider had fraudsters filing fraudulent claimscausing major losses. In particular, fraudsters were using the benefitprovider's IVR system to find valid information, which they then used totake over accounts. In order to detect this kind of probing activity,features were extracted, and a machine learning model was trained on alabeled subset of data and used to determine risk scores for new calls.

IVR Reconnaissance Detection: Example 1

FIG. 13 shows an example of IVR reconnaissance performed by a frauddetection system. A fraudster made nine calls from the same ANI in aspace of ten minutes. The call flow sequence sample shows severalattempts at entering a valid PIN number, followed by a card replacementattempt. This is only a small sample of the call flow sequence, but therest follows a similar pattern.

IVR Reconnaissance Detection: Example 2

FIG. 14 shows an example of IVR reconnaissance performed by a frauddetection system. In this second instance, a fraudster makes multiplecalls from the same ANI in a space of one hour. The fraudster makesseven attempts at entering a valid PAN number, culminating in twosuccessful PAN and PIN number entries.

This use case produced a large amount of data. The state benefitsprovider handles about 20 million calls per month, leading to hundredsof gigabytes of CDR data. Therefore, it may be necessary to scalefeature extraction and machine learning to a high-volume data set.

FIG. 3 is a block diagram illustrating a system for determining a riskscore of a call according to one or more embodiments of the presentinvention. FIG. 3 includes call information from prior calls 305, userfeedback 310, call information data store 315, structured data includingfeatures derived from call information and IVR call flow/queries 320,phone reputation features 325, labeled unique identifiers 330, machinelearning model training unit 340, machine learning model 345, call 350,feature vector 355, comparator 357, and call-specific output/riskscore(s) 360.

Unique identifiers may be in the form of, without limitation, an ANI,originating telephone (phone) number, an account number or accountidentifier (account ID, PIN, PAN), a social security number (SSN), ausername, a subscriber identity module (SIM) authentication key, aninternational mobile subscriber identity (IMSI), or an internationalmobile station equipment identity (IMEI).

Call information from prior calls 305 may be stored in the callinformation data store 315. The call information from prior calls 305may include or be based on IVR call flows and/or CDRs. The callinformation from prior calls 305 may also include, without limitation,user feedback 310, labeled unique identifiers 330, and phone reputationfeatures 325. The call information data store 315 may be implementedusing a relational or non-relational or document-oriented database, andthe database may be managed by a database management system (DBMS). TheDBMS may be MongoDB. The DBMS may be MongoDB in a PyMongo environment.The database may be a graph database.

Labeled unique identifiers 330 may include a unique identifier, such asANI, and a label, e.g. “fraud” or “not_fraud” or “human” or “robot”, inthe form of a (ANI, Label) pair. The user feedback 310 may identifyinstances that are false positives or false negatives and may be in theform of a (ANI, Label) pair. Phone reputation features 325 may include,without limitation, a unique identifier (such as ANI), the callingparty's carrier, a device type (e.g. landline, cell, VoIP, softphone, orspecific phone model), and complaints associated with the uniqueidentifier. Queries 320 are made against the call information data store315, and the queries return structured data including features derivedfrom call information and/or IVR call flow 320.

Labeled unique identifiers 330, user feedback 310, phone reputationfeatures 325, and/or the structured data including features returnedfrom queries 320 (collectively, “training data”) may be passed to themachine learning model training unit 340. The machine learning modeltraining unit 340 learns a machine learning model 345 based on thetraining data.

A feature vector 355 may be extracted from call 350. The feature vector355 may be extracted from the IVR call flow and/or the call's metadata,including a CDR for the call. The feature vector 355 may be passed tothe comparator 357, which compares the feature vector 355 to the machinelearning model 345 to produce output 360 specific to the call. Theoutput 360 may include a risk score for the call. The risk score may bea label from a finite set of labels, e.g. {fraud, not_fraud} or {human,robot}. The risk score may be a numeric value indicating a confidencelevel regarding whether the call can be labeled with one of the labelsfrom the finite set of labels, e.g. a scale comprising integers from 1to 10, with 1 being least likely fraudulent and 10 being most likelyfraudulent.

Some of the calls regarding which call information is stored in the datastore 315 may have no label or risk score associated with them. As such,it may be desirable to determine a risk score for these unlabeled calls.Queries may be made against the call information data store to obtainstructured data 320 for unlabeled calls. This structured data 320 forunlabeled calls may be passed to the comparator 357, which in turnproduces output 360 which may include risk scores for the unlabeledcalls.

FIG. 4 is a block diagram illustrating a method for determining a riskscore of a call received by an Interactive Voice Response (IVR) systemaccording to one or more embodiments of the present invention. First, afeature vector is created (410) based on interaction with the IVR systemduring the call. Second, a machine learning model is used (420) todetermine the risk score of the call based on the created featurevector. Optionally, an action may be taken (430) based on the risk scoreof the call. The feature vector may be based on velocity feature(s) ofthe call, reputation feature(s) of the call, or behavior feature(s) ofthe call.

FIG. 18 is a block diagram illustrating a method for determining a riskscore of a call according to one or more embodiments of the presentinvention. First, information extracted from received calls is stored(1810). The information extracted from received calls may become acomponent of, or an input in the determination of, a reputation feature,a behavior feature, or a velocity feature. Second, queries of the storedinformation are performed (1820) to select data using keys, wherein eachkey relates to one of the received calls, and wherein the queries areparallelized. Third, in at least one embodiment, the selected data aretransformed (1831) into feature vectors, wherein each feature vectorrelates to one of the received calls and includes a velocity feature anda behavior feature. In at least one embodiment different than theembodiment containing step 1831, the third step is as follows: theselected data are transformed (1832) into feature vectors, wherein eachfeature vector relates to one of the received calls and includes avelocity feature and a reputation feature. In at least one embodiment,the third step may be as follows: the selected data are transformed intofeature vectors, wherein each feature vector relates to one of thereceived calls and includes a behavior feature, a velocity feature, anda reputation feature. Fourth, during the call, the risk score isgenerated (1840) for the call based on the feature vectors.

FIG. 19 is a block diagram illustrating a method for determining a riskscore of a call according to one or more embodiments of the presentinvention. First, a call from a particular phone number is received(1910). Second, pre-stored information relating to the particular phonenumber is retrieved (1920) to derive a reputation feature and a velocityfeature. Third, the reputation feature and the velocity feature areincluded (1930) in a feature vector. Fourth, the risk score is generated(1940) for the call based on the feature vector. Optionally, an actionmay be taken (1950) based on the risk score of the call.

The velocity feature of the call may be, without limitation, any of thefollowing features: a number of calls (e.g. from the originating phonenumber with one or more destination phone numbers specified); a durationof one or more calls (e.g. from an originating phone number with one ormore destination phone numbers specified); a duration of time betweencalls or a periodicity between calls; a number of distinct uniqueidentifiers or account identifiers (e.g. the number of account IDsassociated with the originating phone number with respect to one or moreinstitutions, e.g. one or more banks); a number of distinct uniqueidentifiers associated with an account identifier (e.g. a number ofdistinct originating phone numbers associated with an accountidentifier, where the association might exist, for example, in the formof a relation in a relational database or, for example, is made as theresult of a query against a non-relational database); a number ofdestinations called (e.g. a number of destination phone numbers calledby an originating phone number); a sequence of calls or attempted callsfrom at least one originating phone number similar to an originatingphone number of the call (e.g. a sequence of successfully and/orunsuccessfully connected calls from multiple phone numbers sharing anarea code plus first three digits (210-555-6548, 210-555-4961, etc.)with an originating number of the call, wherein the originating numberwas 210-555-1234); or a number of times a specific activity in an IVRwas performed (e.g. a number of times a PIN change or authenticationattempt was made, whether during one call or during a plurality ofcalls).

The reputation feature of the call may be, without limitation, any ofthe following features: suspicious or malicious activity; a priorcomplaint (e.g. complaints lodged by third parties against theoriginating phone number); a device type (e.g. landline, cell, VoIP,softphone, or specific phone model which is the calling phone); acarrier (e.g. mobile carrier, VoIP service provider, or landline or PSTNcarrier of the calling party); a route taken by the call prior toentering a telephone exchange; a route taken by the call after leaving atelephone exchange; or a location (e.g. base station or cell towerinformation).

Behavior features capture how a caller acts during one or more calls,particularly when interacting with an IVR system. The behavior featureof the call may be, without limitation, any of the following features:an action taken during interaction with an IVR system, an amount of timeelapsed between actions taken during interaction with an IVR system(e.g. a timeout after the caller is prompted to enter a uniqueidentifier), providing incorrect information to an IVR system (e.g.multiple incorrect entries when prompted for a unique identifier such asa social security number or a PIN), a number of times a specificactivity in an IVR was performed (e.g. a number of times a PIN change orauthentication attempt was made, whether during one call or during aplurality of calls), a number of times an IVR was called (e.g. aninordinate number of calls from one phone number or one user), a numberof times an IVR was called during a measure of time (e.g. an inordinatenumber of calls from one phone number or one user in one day), asequence of actions taken during interaction with an IVR system (e.g. acaller tried to reset a PIN number an unusually high number of times, ore.g., a sequence of actions in an IVR call flow that may be representedusing an N-gram of high predictive value in a vector space model), avolume or a duration of at least one DTMF tone during interaction withan IVR system, an amount of time elapsed between DTMF tones, a use ofvoice during interaction with an IVR system (e.g. a caller providesspeech input instead of DTMF tone(s) to respond to an IVR prompt), anamount of time elapsed between a beginning of an IVR prompt and a user'sspoken response to the IVR prompt (e.g. whether the user listens to theentire prompt or speaks soon after the prompt begins may indicate priorknowledge of the IVR system's options), or an amount of time elapsedbetween an IVR prompt and an action taken during interaction with an IVRsystem (e.g. an inordinate amount of time between a prompt for a socialsecurity number and a user's response, also e.g., an unusually shortperiod of time between an IVR prompt and a user's response may indicatea robot).

FIG. 5 is a block diagram illustrating a method for generating a machinelearning model which may be used in determining a risk score of a callaccording to one or more embodiments of the present invention. First,call information is inserted (510) into a data store. Second, featurevectors are generated (520) from the call information. Third, thefeature vectors are loaded (530) into a machine learning system. Fourth,a machine learning model is generated (540) using the machine learningsystem. The generated feature vectors may be loaded directly in themachine learning system without first storing them in the data store.Therefore, optionally, the feature vectors are stored (550) in the datastore.

FIG. 7 is a block diagram illustrating a method for determining a riskscore of a call according to one or more embodiments of the presentinvention. First, a feature vector for a call is determined (710).Second, machine learning is used (720) to determine a risk score of thecall based on the feature vector of the call. It may be possible todetermine the risk score of the call in-memory. Storing call informationfor the call in a data store may be optional (705). Further, storing thefeature vector for the call in the data store may also be optional(715).

Example: Calling Card Company

A major calling card company had a problem of premium rate fraud,wherein a bad actor uses stolen calling card numbers to make calls tofake premium numbers in other countries and then pockets the fees. Itwas determined that the bad actors were using automated robots, both todiscover valid calling cards (ANI scanning), and to actually call usingthose cards. By designing features based on graph analysis, along withfeatures capturing duration of calls, interval between successive calls,and periodicity of calls from a source, over 80% of the premium ratefraud was detected, in some cases, up to ten days before actual fraudcalls. A custom feature was created to detect ANI scanning byidentifying sequences of attempted calls using similar ANIs. The customANI scanning feature identified approximately 50% of the premium ratefraud, as shown in the chart (1700) in FIG. 17.

CDR Analysis at Scale

In at least one embodiment of the present invention, Python data sciencetools such as pandas and IPython were used to generate CDR and IVR callflow features. This required that only a sample of the larger dataset beprocessed. MongoDB was used to store CDR data and to query for featuresused to build risk prediction models. This implementation enabled thebuilding of prediction models using more than 100 million records.

MongoDB was chosen based on several requirements. It has a Python driverknown as PyMongo that interoperates with the Python data science stack.It is schemaless or document-oriented or non-relational and thereforecan ingest CDR data with arbitrary formats. MongoDB, like many datastores, provides a bulk insert API that allows for insertion ofthousands of CDRs in a single API call. Finally, MongoDB's AggregationAPI provides a flexible search API that supports query parallelism andhorizontal scalability (“sharding”).

Data Ingest

A key determinant of write throughput is the “write concern” setting.Write concern describes the guarantee that MongoDB provides whenreporting on the success of a write operation. Operationally this meansthe greater the guarantee, the slower the insertion throughput. The“journaled write concern” was used for bulk insert operations toguarantee that CDRs are fully committed to disk after each insert. Forupdates, the “acknowledged write concern” was used, which simplyguarantees the database server received the request. For a single nodeMongoDB cluster and the aforementioned settings, insert speeds of 8000records per second and updates of 1000 records per second were realized.

Generating CDR and IVR Features

FIG. 10 is a block diagram illustrating a method for generating featuresaccording to one or more embodiments of the present invention. First, afield of unique identifiers is selected (1010) from a data store.Second, at least one additional field is selected (1020) from the datastore. Third, objects are generated (1030), wherein each of the objectsincludes a value from the field of unique identifiers and any valuesfrom the at least one additional field related to the value from thefield of unique identifiers. Fourth, the generated objects are queried(1040) for features. Each unique identifier may be related to one ormore values from the additional field, and feature vectors may comprisethese values along with their related unique identifier. Optionally,before the field of unique identifiers is selected from the data store,data in at least one field of unique identifiers may be represented(1002) using action words, and the data may be stored (1002) in a datastore. Optionally, before the field of unique identifiers is selectedfrom the data store, data in at least one additional field may berepresented (1004) using action words, and the data may be stored (1004)in a data store. A supervised classifier may be defined in a featurespace that includes the dimensions of the at least one additional field.

FIG. 11 is a block diagram illustrating a method for generating featuresaccording to one or more embodiments of the present invention. First,interaction with an IVR system is represented (1110) using action words.Second, the action words are converted (1120) to a feature vector. Asupervised classifier may be defined in a feature space that includes atleast some of the dimensions of the feature vector.

In at least one embodiment of the present invention, feature vectorsincluding behavior features may be collected from a plurality of priorcalls from a same phone number. The feature vectors including behaviorfeatures from the plurality of prior calls from the phone number may beused to train a machine learning model or fingerprint representing thephone number. A call from the phone number may be converted to a featurevector that may be compared against the machine learning modelrepresenting the phone number to determine a risk score for the call, orthe call may be combined with other calls to form a feature vector thatmay be compared against the machine learning model representing thephone number to detect a malicious actor.

In at least one embodiment of the present invention, feature vectorsincluding behavior features may be collected from calls having a certainlabel from a plurality of phone numbers. The certain label may beobtained from user feedback, including feedback regarding a falsepositive or a false negative. The certain label may be added by humansupervision, including expert supervision. The certain label may be alabel from a finite set of labels, e.g. {fraud, not_fraud} or {human,robot}. The feature vectors including the behavior features may be usedto train a machine learning model representing the certain label. A callmay be converted to a feature vector that may be compared against themachine learning model representing the certain label to determine arisk score for the call.

In at least one embodiment of the present invention, feature vectorsincluding behavior features may be collected from labeled calls, whereinat least one of the labeled calls has a first label and at least one ofthe labeled calls has a second label. The first label may representgenuineness, including human, not fraudulent, and/or not malicious, andthe second label may represent non-genuineness, including robot, fraud,and/or malicious action. The feature vectors including behavior featuresmay be used to train a machine learning model representing the firstlabel and the second label. A call may be converted to a feature vectorthat may be compared against the machine learning model representing thefirst label and the second label to determine a risk score for the call.

The Aggregation Framework provides a flexible API to query for IVR andCDR features used to build risk prediction models. FIG. 8 shows a simplequery to count the number of unique accounts associated with aparticular phone number.

Aggregation queries are constructed as a data processing pipeline. Line2 selects “ANI” and “ACCOUNT_ID” fields from the records. Lines 3-7generate group objects where the key is “ANI” and the value is a set ofunique account numbers. Line 8 generates a Cartesian product between“ANI” and “ACCOUNT_ID”. Finally, line 9 generates group objects wherethe key is “ANI” and value is COUNT of “ACCOUNT_ID”.

The PyMongo API is thread-safe, and therefore the Aggregation queriesmay be parallelized using multiprocessing.dummy.pool. The pooldistributes queries into groups of ANIs to be queried by each threadconcurrently. Each added thread provided a linear reduction in querylatency.

Storing Feature Vectors

The IVR and CDR features collected from queries may be used to generaterisk prediction models using scikit-learn. MongoDB may be used to storenormalized feature vectors as binary JavaScript Object Notation (BSON,i.e. binary JSON) data. This allows prediction processes to reuse thefeature vectors for subsequent experiments.

FIG. 9 demonstrates how to store feature vectors to MongoDB in a PyMongoenvironment. Lines 4 and 6 of FIG. 9 serialize a document “X” includingn training samples and a document “y” including n labels, which arerepresented as binary data via lines 3 and 5 and then stored to acollection “cdr” in line 2.

An action may be taken based on the risk score for a call. The actiontaken may be displaying, during the call, the risk score on a displaysuch as a monitor, a display on a phone, a smartphone, or a tablet. Theaction taken may be storing the risk score in a database during thecall. The action taken may be altering, during the call, an IVR callflow based on the risk score. For example, if the call were deemedfraudulent, the IVR may ask the caller to update an address. The actiontaken may be notifying the police or an owner of an IVR system inresponse to the risk score. The action taken may be disabling a featurein an IVR system during the call in response to the risk score. Forexample, a PIN change menu or a withdrawal option may become unavailableto the caller. The action taken may be locking down an IVR system inresponse to a risk score. That is, the IVR system may no longer respondto any calls or may no longer offer certain options like PIN change orwithdrawal to any callers. The lock down may continue until the fraud isinvestigated and/or a determination is made as to how the fraud wasperpetrated. The action taken may be requiring alternativeidentification during the call in response to a risk score. For example,answers to security questions, a PIN number, an address, a driverlicense number, or a social security number may be requested. The actiontaken may be requesting additional information during the call, and theadditional information may be used to create or update a profileconcerning the caller or the phone number of the caller.

FIG. 6 is a high-level block diagram of an example computer (600) thatis arranged call detail record analysis to determine a risk score for acall and to identify fraudulent activity and/or for fraud detection inIVR systems. In a very basic configuration (601), the computing device(600) typically includes one or more processors (610) and system memory(620). A system bus (630) can be used for communicating between theprocessor (610) and the system memory (620).

Depending on the desired configuration, the processor (610) can be ofany type including but not limited to a microprocessor (μP), amicrocontroller (μC), a digital signal processor (DSP), or anycombination thereof. The processor (610) can include one more levels ofcaching, a processor core, and registers. The processor core can includean arithmetic logic unit (ALU), a floating point unit (FPU), a digitalsignal processing core (DSP Core), or any combination thereof. A memorycontroller can also be used with the processor (610), or in someimplementations the memory controller can be an internal part of theprocessor (610).

Depending on the desired configuration, the system memory (620) can beof any type including but not limited to volatile memory (such as RAM),non-volatile memory (such as ROM, flash memory, etc.) or any combinationthereof. System memory (620) typically includes an operating system(621), one or more applications (622), and program data (624). Theapplication (622) may include a system and method for call detail recordanalysis to determine a risk score for a call and identify fraudulentactivity and for fraud detection in IVR systems. Program Data (624)includes storing instructions that, when executed by the one or moreprocessing devices, implement a system and method for call detail recordanalysis to determine a risk score for a call and identify fraudulentactivity and for fraud detection in IVR systems (623). In someembodiments, the application (622) can be arranged to operate withprogram data (624) on an operating system (621). Program data (624)includes call information data (625). Call information data (625) mayinclude uninitialized variables and arguments to parameters for methodsappropriate to implement the systems and methods described herein.

The computing device (600) can have additional features orfunctionality, and additional interfaces to facilitate communicationsbetween the basic configuration (601) and any required devices andinterfaces, such non-removable non-volatile memory interface (670),removable non-volatile interface (660), user input interface (650),network interface (640), and output peripheral interface (635). A harddisk drive or solid state drive (SSD) may be connected to the system bus(630) through a non-removable non-volatile memory interface (670). Amagnetic or optical disk drive may be connected to the system bus (630)by the removable non-volatile interface (660). A user of the computingdevice (600) may interact with the computing device (600) through inputdevices such as a keyboard, mouse, or other input peripheral connectedthrough a user input interface (650). A monitor, printer, speaker orother output peripheral device may be connected to the computing device(600) through an output peripheral interface (635) in order to provideoutput from the computing device (600) to a user or another device.

System memory (620) is an example of computer storage media. Computerstorage media includes, but is not limited to, RAM, ROM, EEPROM, flashmemory or other memory technology, CD-ROM, digital versatile disks (DVD)or other optical storage, magnetic cassettes, magnetic tape, magneticdisk storage or other magnetic storage devices, or any other mediumwhich can be used to store the desired information and which can beaccessed by computing device (600). Any such computer storage media canbe part of the device (600). One or more graphics processing units(GPUs) (699) may be connected to the system bus (630) to providecomputing capability in coordination with the processor (610),especially where single instruction, multiple data (SIMD) problems arepresent.

The computing device (600) can be implemented as a portion of asmall-form factor portable (or mobile) electronic device such as a cellphone, a smart phone, a personal data assistant (PDA), a personal mediaplayer device, a tablet computer (tablet), a wireless web-watch device,a personal headset device, an application-specific device, or a hybriddevice that include any of the above functions. The computing device(600) can also be implemented as a personal computer including bothlaptop computer and non-laptop computer configurations. Additionally,the computing device (600) may operate in a networked environment whereit is connected to one or more remote computers over a network using thenetwork interface (650).

The foregoing detailed description has set forth various embodiments ofthe devices and/or processes via the use of block diagrams, flowcharts,and/or examples. Insofar as such block diagrams, flowcharts, and/orexamples contain one or more functions and/or operations, it will beunderstood by those within the art that each function and/or operationwithin such block diagrams, flowcharts, or examples can be implemented,individually and/or collectively, by a wide range of hardware, software,firmware, or virtually any combination thereof. Those skilled in the artwill appreciate that the mechanisms of the subject matter describedherein are capable of being distributed as a program product in avariety of forms, and that an illustrative embodiment of the subjectmatter described herein applies regardless of the particular type ofnon-transitory signal bearing medium used to actually carry out thedistribution. Examples of a non-transitory signal bearing mediuminclude, but are not limited to, the following: a recordable type mediumsuch as a floppy disk (620 c), a hard disk drive (620 b), a solid statedrive (SSD) (620 b), a Compact Disc (CD) (620 c), a Digital Video Disk(DVD) (620 c), a Blu-ray disc (BD) (620 c), a digital tape (620 c), acomputer memory (620 a), etc.

With respect to the use of substantially any plural and/or singularterms herein, those having skill in the art can translate from theplural to the singular and/or from the singular to the plural as isappropriate to the context and/or application. The varioussingular/plural permutations may be expressly set forth herein for sakeof clarity.

Thus, particular embodiments of the subject matter have been described.Other embodiments are within the scope of the following claims. In somecases, the actions recited in the claims can be performed in a differentorder and still achieve desirable results. In addition, the processesdepicted in the accompanying figures do not necessarily require theparticular order shown, or sequential order, to achieve desirableresults. In certain implementations, multitasking and parallelprocessing may be advantageous.

What is claimed is:
 1. A computer-implemented method for determining arisk score of a call received by an Interactive Voice Response (IVR)system, the computer-implemented method comprising: creating a featurevector based on interaction with the IVR system during the call; andusing a machine learning model to determine the risk score of the callbased on the created feature vector.
 2. The computer-implemented methodof claim 1, further comprising: representing prior calls as featurevectors; labeling the feature vectors representing the prior calls; andtraining the machine learning model using the labeled feature vectors.3. The computer-implemented method of claim 1, wherein informationregarding the call is stored in a database before the using a machinelearning model to determine the risk score of the call based on thecreated feature vector.
 4. The computer-implemented method of claim 3,wherein information regarding the call is retrieved from the databasebefore the using a machine learning model to determine the risk score ofthe call based on the created feature vector.
 5. Thecomputer-implemented method of claim 1, wherein the created featurevector is based on a behavior feature of the call, and wherein thebehavior feature of the call is a sequence of actions taken duringinteraction with an IVR system.
 6. The computer-implemented method ofclaim 1, wherein the created feature vector is based on a behaviorfeature of the call; and wherein the behavior feature of the call is atleast one of an action taken during interaction with an IVR system, anamount of time elapsed between actions taken during interaction with anIVR system, providing incorrect information to an IVR system, a numberof times a specific activity in an IVR was performed, a number of timesan IVR was called during a measure of time, a volume or a duration of atleast one DTMF tone during interaction with an IVR system, an amount oftime elapsed between DTMF tones, a use of voice during interaction withan IVR system, an amount of time elapsed between a beginning of an IVRprompt and a user's spoken response to the IVR prompt, or an amount oftime elapsed between an IVR prompt and an action taken duringinteraction with an IVR system.
 7. The computer-implemented method ofclaim 2, wherein information regarding the prior calls is stored in anon-relational database.
 8. The computer-implemented method of claim 2,wherein the prior calls are all from a same phone number.
 9. Thecomputer-implemented method of claim 2, wherein the prior calls are froma plurality of phone numbers, and wherein each of the labeled featurevectors used to train the machine learning model has a same label aseach of the other labeled feature vectors used to train the machinelearning model.
 10. The computer-implemented method of claim 2, whereinthe labeling the feature vectors representing the prior calls is doneusing a first label representing genuineness and a second labelrepresenting non-genuineness.
 11. The computer-implemented method ofclaim 2, wherein the representing prior calls as feature vectorsincludes representing interaction with an IVR system using action words.12. The computer-implemented method of claim 2, wherein the representingprior calls as feature vectors includes: representing interaction withan IVR system using action words in a document; and converting thedocument to a feature vector.
 13. The computer-implemented method ofclaim 2, wherein the representing prior calls as feature vectorsincludes representing interaction with an IVR system using at least oneof N-grams or skip N-grams.
 14. The computer-implemented method of claim2, wherein the representing prior calls as feature vectors includesdetermining an inverse document frequency.
 15. The computer-implementedmethod of claim 1, further comprising taking an action based on the riskscore of the call.
 16. The computer-implemented method of claim 15,wherein the taking an action based on the risk score of the callincludes at least one of displaying the risk score on a display duringthe call, storing the risk score in a database during the call, alteringan IVR call flow during the call, notifying police, or notifying anowner of an IVR system.
 17. The computer-implemented method of claim 15,wherein the taking an action based on the risk score of the callincludes at least one of altering an IVR call flow during the call,disabling a feature in the IVR system during the call, locking down theIVR system, requiring alternative identification during the call, orrequesting additional information during the call.
 18. Thecomputer-implemented method of claim 1, wherein the interaction with theIVR system includes responses to the IVR system represented using actionwords.
 19. A computer-implemented method for determining a risk scorefor a call, the computer-implemented method comprising: storinginformation extracted from received calls; performing queries of thestored information to select data using keys, wherein each key relatesto one of the received calls, and wherein the queries are parallelized;transforming the selected data into feature vectors, wherein eachfeature vector relates to one of the received calls and includes avelocity feature and a behavior feature; and generating, during thecall, the risk score for the call based on the feature vectors.
 20. Thecomputer-implemented method of claim 19, wherein each feature vectorincludes a reputation feature.
 21. The computer-implemented method ofclaim 19, further comprising: training a machine learning model usingthe feature vectors; using the machine learning model to generate therisk score for the call; and displaying the risk score for the call on adisplay during the call, wherein the queries are parallelized using athread pool.
 22. An apparatus that determines a risk score of a callreceived by an Interactive Voice Response (IVR) system, the apparatuscomprising: at least one processor; and a non-transitory computerreadable medium coupled to the at least one processor havinginstructions stored thereon that, when executed by the at least oneprocessor, causes the at least one processor to: create a feature vectorbased on interaction with the IVR system during the call; and use amachine learning model to determine the risk score of the call based onthe created feature vector.
 23. The apparatus of claim 22, wherein thecreated feature vector is based on a behavior feature of the call, andwherein the behavior feature of the call is a sequence of actions takenduring interaction with an IVR system.
 24. An apparatus that determinesa risk score for a call, the apparatus comprising: at least oneprocessor; and a non-transitory computer readable medium coupled to theat least one processor having instructions stored thereon that, whenexecuted by the at least one processor, causes the at least oneprocessor to: store information extracted from received calls; performqueries of the stored information to select data using keys, whereineach key relates to one of the received calls, and wherein the queriesare parallelized; transform the selected data into feature vectors,wherein each feature vector relates to one of the received calls andincludes a velocity feature and a behavior feature; and generate, duringthe call, the risk score for the call based on the feature vectors.